'DdCUHEIT BESUHS 



ED 123 410 



CE 007 148 



AOTHOE 
TITLE 

P.DB DATE 
NOTE 

EDBS PRICE 
DESCEIPTORS 



ABSTEiCT 



Farmer, James A.., Jr. 

Evaluating Broad- Aimed Programs of Adult 

Education. 

23 Mar 76 • 

15p. 



I!F-$0.83 HC-$1,67 Plus-^ostaoe,. 
♦Adult Education Programs; ^Educational Needs; 
♦Evaluation Criteria';; ♦Evaluation Methods; 
♦Evaluation Needs; ' *P^ogra« Evaluation 



The document examines the need to develop, and 
implement more ade^g^ate. evaluation aethoUs for broad-aiaed adult 
education progr,a'ars, and presents some suggestions. Broad^-aiM^d adult 
education fjrpgrams are 1 described as those which seek to affec<the 
quality oj 'the learners' lives through education and to help le^^ers 
cope more effectively yith their problems. Thii? type of program is" 
generally /funded^- under the Smith lever Act, 191^1, the Manpower 
Development and Trailing Act, 1962, Title I of Higher Education 
Actr 1965', and the Ad.ult Education Act, 1966. The , paper «s "four 
sections, include: (1) broad-aim^d adult education defined and 
differentiated froa other types of adult education/. with three 
program types examined; (2) need for more adequate evaluation of 
*brl5^d^EriX<^"a^d¥rt^d^^ pirogfams, eljcaainifig curri^nt evaTua'tiDn 

techniq.ues; (3| false assumptions to be avo^ided when evaluating 
broad^-aimed programs, giving examples of inappropriate evaluation 
methods; ^ and, (4) toward more adequate evaiiiation of broad-aimed 
adult education programs, describing one approach and presenting 
various opinions. (IH) y 
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* VMiv^i 1 EDUCATION POSITION OR POLICY 

Introduction 

Currently, not only In this country but'also In almost every other part" 
of t'he world, increased attention is* beir>g, paid to adult education. After 
many years of being in a marginal position, adult education has come into the 
Umelight^ Being in the limelight has its advantages but also its costs. 
.Initially, what comes into th'e limelight tends to receive attention, at least . ^ 
InVpart, because of Its novelty. Sooner or later, however, what remains In 
the limelight more than^ momentar i ly usually gets closer scrutiny. It needs 
to be understood-and- I ts worth- needs to be established. 

Adult education, in its many forms ranging f romxfunctional literacy and 
adult basic education to continuing professional educat io^i, has been in the 
limelight long enough for the novelty, for the most part, to have worn off . 
We are well into .the period when it needs to be understood and its worth 
establ'ishea. 

Focus of This Paper 

Many of the adult education programs that have received the greatest 
amount of .national and international attent.ion and funding are relative. y 

V'broad-aimed" (Weiss and Rhein, 1969)\ in contrast to adult education pro- 
grams with relatively narrow aims—such as those seeking pr'imarily to teach 
something with little concern.for how what is learned Is l^ter used-broad- 

• aimed programs seek to affect the quality of the learners' lives 
through education and "to help the learners cope more effectively with their 
problems (Farmer, 1975b, p. 23). Included in those adult education programs 
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ihot are relatively broa^J-almed are those that have been funded under the 
Smith Lever Act of 1914/{ the Manpower Development and Training/Act of 1962, 
Title I of the Higher/Education Act of 1965, and the Adult Education Act of 

1966, Elsewhere in /the world, many Functional Literacy programs are explicitly 

/ - ^ 

broad-aimed, since/ they seek not only to teach literacy, but also to affect the 

/ ' :< ( ! 

quality of the learners* lives and to help them cope more effectively with 
their problems and those of the communities in which they live. 

The number and s'^ze of broad-aimed adult education programs have increased 
rapidly during the /past few years. Unfortunately, developing and implementing 
manageabley feasible, and ''situation specific" ways tq evaluate such programs 
have been /largely neglected. It would seem that valid summative evaluation 
is needed to provide information essential to determining the worth of 
these p^rograms. 'Valid formative evaluation is needed to produce Information 
that can be*- feedback during further developmeni of broad-aimed adult educa- 
tioo /programs to improve them. / * • . 



in , the current paper, attention is focused on broad-aimed adult education. 

^nc^' particularly on ways to evaluate it. To do so: 

\. Broad-aimed adult education is further defined and differentiated . 
froifi other types of adult education. 

2. Evidence of the need for more adequate evaluation of broad-aimed 
adult education programs is descrTbed. 

5. Several false assumptions to be avoided in evaluation of broad-a/me<i 
adult education programs are presented. 

^. Away to conceptualize and Implement more adequate evaluation ^of 
broad-^aimad adult education programs Is suggested. 

8road-Aimed Adult Education Defined and' Oi f ferentiated from Othe/ Types 
of Adult Education V 



As indicated above, inroad-aimed adult education seeks not only to teach 
something but also to affect the quality of the learners* lives through 



education and help them cope more effectively with their problems. It has 

bdcn found helpful not only in conceptual i zi ng and ^ifnplementing education 

for adults but also for evaluating it, to distinguish between the following ^ 

\ . . .. 

types of adult education 

I. Type I— Content Centered Adult Education . This type of adult 
education is provided primarily to teach knowledge, attitudes, or skills. 
This educational process starts wi th what is to be taught and who is to 

teach it. Then,, learners are sought who are wjlUng to learn that type of 

information. 

2^^ Type ll--Learner Centered Adult Education . In this type of adul^^ 
"education, attention is pal d. primari ly to assisting adults in learning what 
it is they wish to learn, usually with the teacher acting as a facilitator 
who helps as a co-investigator or as one who R^ke possible self-directed 
learning. The learner typically seeks to learn particular knowledge, 
attitudes, or skills, which will help in solving problems important to himself 
artd, hopefully, in solving similar problems on his own. 

3. Type lll--?robiem Centered Adul t Education . This type of adult 
education primarily focuses on problems that require some form" of learning 
•in order for the problems addressed to be solved, this approach starts with 
5 real and pressing problem and asks: "What is it in the solving of this 
proo^em to wHich the educat ibn of adults can contribute through the learning 
of knowledge, attitudes, or skills?" Then the question is asked: "Which ■ 
adults need to be and can be involved in educational experiences wr^^ch car. 
lead or contribute to the solution of the problem addressed?" {Farmer, 1974, 
. p. ^3). 
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Broad-aimed adult education is a type of problem-centered adult educa- 
tion (Type Ml:), which focuses mainly on community or societal, j^prob I ems. In 
broad-aimed adult education: 

K Th^re is evidence of its explicitly addressing community or 
societal problems by Unking educational, resources to those 
problems ; 

2* The problem that is addressed Is significant in terms of its 
.size, complexity, and the extent of the peed (i.e*, suffering, 

etc.)'; • • 

" r , * • 

3; Its approach is educational rather than manipuliative or 'direct 
action-oriented ; ^ 

k. There is a strong ikelihood that what is learned in the educa- 
tion provided will result In more effective community or societal 
problem solving efforts on the part of the learners arid that those 
improved efforts, in turn, will make a positive contribution to 
the amelioration of community or societal problems. (Farmer and 
Knox, 1976, p. 280' 

In other words, broad-aimed adult education is a form of education that has 

been deliberatly linked to community or societaffiroblem solving. 

Some forms of education are generally viewed as being primarily of 

intrinsic value ("having value in themselves, for theirown sake, and not as 

a means only'' [Runes, 19V» P- ^^8]). Other forms of education are provided 

basically becau^S" they are thought to be of instrumental value ("having value' 

, •' ' . - * ^ 

due to the useful consequences which they produce, a value as a means, a value 

as a contribution" [Runes, 19^2, p. 330]). 'Broad-aimed adult and continuing 

education programs are designed to be mainly of instrumental value. -Arter^-a 



part icular ! broad-aimed education program has been itnpleniented » I ts i nscrjjmental 

■ ' ■ r • 

value can be assumed to have been established to the^ extent that, problem 
solving efforts have been improved , as a result of what was learned in the educa- 
tion provided, and in turn, that community or societal problems were ameliorated 
as a result of those efforts, /n those instances when a broad-aimed educational 
program has been developed an/ implemented but no evaluative evidence concerning 
resulting changes In proble^ solving efforts of the learners and'the effects 

/ 



on,. the amelioration of the" problems has been obtained, the program can be ' 
considered to be ''truncated*', (Farmer and Knox, 1976, pp. 28-29) 

2. Need for. More Adequate Evaluation of Broad-Aimed Adult Education Programs 

Both in the United States (Caro, 1970) and elsewhere (Coombs, 1973), 
the major focus of broad-aimed adult andl conti nuing education programs has 
been on action, with little or no attention being paid to evaluation^ Based 
o,n a survey of admi nist rator^ of continuing education programs in the mid- 
western United States, the following reasons for lack of attention having 
been given to the evaluation of their programs: 

1. insufficient pressure for program accountability. 

2. Inadequate understanding of program evaluation^ and of ways to 
conduct validly and feasibly. 

3. Reluctance to use money, time and/or other resources oh pro- 
j gram evaluati on. 

• ^. Unwillingness to require or even ask that clients take the 
' ^ -rte'cessary time to provide evaluative feedback. 

5. Reluctance to learn evaluative results. 

6. Feeling that determining the worth of a program can be done 
adequately merely on a subjective and impressionistic basis* 
(Green, 197^; Pennington, ^37^). 

According to Peters (1975, P- 3)7) and WiMiams (197^*, P- 137), evalua- 
tions of broad-aimed educational programs for adults that have been conducted 
.have tended to rely heavily on the following types of Indicators of success: 

K Number of '^graduates**; 

2. Number of students placed in joos; 

3. Number of participants; 

(Occasionally), the gain in achievement scores made by adult learners; 
5. Degree of participation; 
6* Level of involvement; 

■ \ 6 



7.\Gradual or rapid .expanlon of a project to reach more of an 
Intended pojSulation; -- \ 



8, Number of instructors trained since the beginning of a project; 

9, The rate of learners' dr'oppincjrout of classes; 

to. The degree to which the objectives set out by a program have 
been achlevied, , , • 

To be sure, from some role perspectl ve(s) (l,e., as viewed by the fundcrs 
program administrators, or teachers) positive findings regarding these 
Indicators, individually or in combination, can be taken to suggest' that pro- 
jjress has been made in implementing a broad-aimed educational program and 
that that implementation has seemingly had some effect on the learners, A 
broad-aimed education program for adults can achieve all of the above,, how- 
ever, and stin not have a positive Impact on the lives of the learners and 
on the society about them. The worth of a broad-aimed educational program 
for adults hinges on the nature and extent of Its success in-reducing 
community or societal problems, or in other words, of improving quality of 

V 

life, (Farmer, 1975a," p, 172) 

in describing the need for more adequate evaluation of federally funded, 
broad-aimed adult education program, one agency stated: 

The funding of the (broad-aimed educational program for adults) 
has been carried out over the past five years without adequate 
assessment of the magnitude or persistence of the effects of the 
program upon the State in general or, more specifically, upon the 
institutior^s receiving the funding or the communitjes to be served. 
Neither the\jua«"terly progress reports- nor the self-evaluatiye nnai 
reports fromUhe funded institutions, nor the onsite visits .rom ^ 
the agency's Vdministrator^- fs sufficient in itself or in combination, 
to provide an^objective measure of the benefits of this federal pro- 



gram } 



The l^ck of (adequate) evaluation of the program, a lack wht.ch exists 
not only in this State, but nationally, has prolonged the unfortunate 
situation In which the .personnel have been forced to continue making 
decisions without the benefit of sufficient feedback, /(The Request for 
Proposal for the Evaluation of Title 1 (HEA,^1965) Programs In 
California, 1965-1972, The California Coordinating Council for Higher 
Education, Sacramento-, 1972). 
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3. Folso Assumptions lo bo Avoided 

When the decision is made, either under externa' 1 mandate or voluntarily, 
to. evaluate a broad-aimed educational program for adults., care needs to be 

•'V 

taken to identify and avoid.wha't, for thcs'type of evaluation, can be con- 
sidered false assumptions: , 

1. that more ts necessari ly .better 

2. that the use of evaluative tools such as testing, participation, 
and Interviewing, In arid of Itself, constitutes valid program 
cvoluatlon; 

3. that evidence that a broad-aimed program has achieved relatively 
narrow objectives necessarily establishes its worth as a broad- 
aimed program; 

/ i». that no types of causal relationships can be established between 
what participants learn in broad-aimed educational programs, 
how they apply that knowledge in. social problem-solving, and the 
consequences of -that appl Ication; 

5. that the reduction of a gap between what the funders or admini- 
strators intend and what happens In the_Brogram is necessarily 
positive; /\ ^ * 

6 that a valid evaluation of a bifoad-aimed education program for 
- adults can be accomplished by focusing merely on components i)f- 
the program; without regard to the program as a whole or th6 
ways that It interrelates with Its environment. (Farmer, 
pp. 25-26) . 

Building program evaluation on such f^lse assumptions is likely to 
distort significantly the evaluative findings. Using distorted feedback 
may well harm the programs that are evaluated. 

^ Evaluating a broad-aimed, problem-centered adult education program 
(Type III) as If it were a content-centered (Type I) or a learner centered 
(Type II) adult education program would be like evaluating a water bed as if ij^ 
were a standard bed (equipped with a frame on legs, a set of springs,, and a 
mattr.ess)_pr a traditional Japanese bed (consisting of a mattress placed on 
floormats.) _ 
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Evaluating a brood-aimed adult education program as if it were 'a 
-narrow-aimed one, moreover, would be similarly inappropr iate. Doing so 
.can^i-^ad to erroneous conclusions, similar to those that would be reached 
by evaluating the following *story as a narrow-aimed event. 

A sportsman in Illinois decided to go moose hunting in Hinnesota 
on his vacation. He mode reservations by telephone for a guide 
to take him moose hunting. 

When the time came to leave for Minnesota, the sportsman put his 
guns in the back of his jeep. Just before he left, however, he went 
back into the house, and got his fishing equipment, and put it also 
in the jeep. 

Upon arrival* in Minnesota, he learned that there were no moose in the 
vicinity. Talking to persons in the sports store, however, he learned 
that muskie, an^ exciting game fish, were hitting. He cancelled his 
reservation for the moose guide, hired a muskie guide, and day after 
day caught a .limit of near-record sized muskie. 

After his vacation, he wrote up the story and sold U to a sport^ 
magazine. / 

Question ff\ : How would you evaluate the vacation as a moose ^f^^nt^ 

(the specified narrow aim or objective of the vacation)? 

If the vacation were to be evaluated mainly as a moose hunt, one would 

most likely conclude that it was a failure, since no moose were shot. 

Question ^2 : How would you evaluate the vacation as a broad-aimed 
event? 

Evaluated as a broad-aimed* event, the vacation clearly was a success, 
since: 



U\ The sportsman showed a good foresight iin bringing along several 

I types of sports equipment^ .broadenening the » chances of his having 
^^-^1 a successful vacation. , 

i / \ ■ 

2. ■ He^^evidenced flexibility in shifting f^rom moose hunting to (Dusk^e 

fishing when the circumstances warranted. 

3. The fishing was excellent. \^ 

He obtained an unintended and unanticipated side-benefit from 

the money received ftom selling the story. / 

— — ~* ^ ^ ^ ^ 

5. No_uadesirablc events or side-effects, reportedly occurred. 

'9 - ■ • 



. All in all, It seems to have" been an ex.cellent vacation, a conclusion 
completely opposite from the one i^rrived at by evaluating the vacation 
invali dly as a narrow-aimed event • ^ ' 

* Unfortunately, all too frequently broad-aimed adult education programs 
have been evaluated (equally InvaUdly) as if they were narrow-aimed pror 



grams. 



Toward More Adequate Evaluation of Broad-Aimed Adult Educat ion Programs 
During the past*^few years, efforts (Farmer, Sh^ats, and Deshler, 1972; 
Farmer, 197^; Farmer and Papagiannis, 1975) -have been made^ to conceptualize 
.an,a implement more adequate ways of evaluating broad-aimed adult education 
programs. In essence, the approach that Has be|^en developed: 

K reflects a. broader-aimed evaluation. approach than is implied 



in most other evaluative ef 



arts dealing with broad-aimed 



educational programs for adults. 

2. is concerned with the evaluative needs not only of those who 
.are providing the program h/ut also of the learners, instructors, 

funders,,and other client groups who are interested in the pro- 
gram being evaluated. 

3. deals with'broad-aimed educational programs for adults as a 
chain of events, starting with the initial exposure to an idea 
to* the actual implementation of that idea and the consequences 
of that implementation- Because broad-aimed educational pro- 
grams for adults is generally viewed as being a purposive 
type of education, designed to serve useful ends or functions 
though not necessarily as a result of deliberate design, it 

. seems appropriate cp focus attention not only on intended 

consequences but also on unintended but anticipated consequences^ 
nnd uninlonddil, unanLici paU^d coni;o<|ii.oncoK- 

X emphasizes the importance oT looking at-environmcntal constraints 
that affect the implementation of broad-aimed educational pro- 
grams for adulU, problem-solving efforts of learners as a result 
of such programs\* and the impact of 'those efforts 



Attempts to consider all aspects of a broad-aimed adult education 
program can lead to the Identification of virtually.an endless number of 
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cvalLuivc cuicslions. Afler having become overwhelmed by evaluatiye 
questions, one- group of eValuators observed: ... 

The question-generating approach got wholly out of hand. Questions 
/ multiplied like the proverbial rabbits while the members of the 
i losk force hjokod on'wi Lh M.jllhusian f'»*el>f«li rMj. Aliio, the quest lon- 
gcneraLim>»>i^oGcs's, in which wo found ourselves cnsnorU'.d, seemed 
more circular than linear i n ,di rection. First, we engaged in" 
fission and smashed big questions into numerous subquestions. Next, 
reacting In dismay to what we had wrought, we fused subquestions 
into major questions which closely Vesembled their forebears of 
two generations ago. It is not to be denied that futile tasks 
can be Instructive, but it is our duty to urge others to seek a 
different conveyancy rather than" mounting our treadmill. 

!n a recent\1y publt shed monograph on Program Evaluation (Farmer and 
Papaglannis, 1975), the author of jhls paper has concluded: 

Whatever conveyance Is iised, it needs to lead"to program evaluations 
that are manageable, feasible) and situation- specif tc; that portray 
tJhe wholeness ofprograms and \the relationship between them and 
their environments; and that pVovlde a defensible basis for 
deciding exactly what Is needed. ' \ 

Ho prefabricatecl approach, no eyaluatiye model or design,, or any 
set of general /evaluative questi,ons can provide the type of 
evaluations .that have all the ab*pve cl^ar^cter 1st i.cs. Rather, a 
."decisipn-making" approach is ca|led for. In this appr(j)ach, 
persons associated with the program, along. with one or. more 
external eva'luators and possibly V representative of those who help 
fund the programs collaborate i,n planning and conduct aj specific 
Vevaluation effort that takes into Recount each of the foregoing, 
tend i.t ions. > 

Looking at only a few isolated components of a program jis an invalid 
approach to evaluation. Yet under most circumstances, jevalua ting 
. a whole progronAin great depth Is not feasible. One effective tool 
in the aeci s ion -making process, might 'be what Amitai Etzioni calls 
"mixed scanning," which entails scanning a program andj Its environ- 
Miient much as an infantry scouts scans a battlefield under fire. The 
scout rapidly views his strategic position, identifies potential 
danger or opportunity, spo ts, and gi vesthem close scrutiny. He 
does .^ot have time to gatnerall the facts, nor can he look at oni.y 
what \s in frofit of him. His task is to scan, assign priorities, 
and select those areas that need careful .scrutiny. Failure to do 
this may well result In a missed opportunity or a costly mistake. , 
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The following components will typically need to be scanned: 

1. .need for the program '/ 

2. philosophical considerations 

3. values * -p- 

k. assumptions underlying or otherwise related to the program ' 

I 

5. the degree olp the program's development, in general and In , 
local situations- • ' ^ / 

6. the context or environment in which the program functions - 

7. aljternative ways that the prograrji^ Has been and Is^being 
im|>lemented 

8; consequences ^of the prpgram 

9. explanations of consequences-rthe extent to which those 
consequences ha^e been jattributed to the program 

' ! ' - . 

The effectiveness of a program is judged not dnly by each of these 

components, but also, and perhaps more importiantly, by the- relation- 
ship between them. / 

If little or nothing IS known about the results of the program, 
major attention. might well focus on results'. If however, there 3s 
ample evidence of the nature and extent of the results and reason 
to think that some of its components are functioning unsatisfactorily, 
then major attention might better focus on the ^educational process/ 
on the program inputs from learners, t^eachers, and administrators, 
ar|d on the program* s context. 

In this decision-making, process, care ned^s to be taken not to 
identify as critical so. many aspects that ^the evaluative effort 
becomes Swamped with too many details. At the same time, the 
evaluator\i 1 1 want to examine the aspects i^elected from the point 
of view of the program as a whole— how they fit into and affect 
the program, ^nd how much they contribute to the overal ixworth 
of the program. (Farmer, I975b,^ pp. 29-311 \ 

In gathering evaluative data concerning^r6ad-aljin\ed adult education 
programs, the use of what Oenter (1970| has desjfribed as ••elite and 
specialized Interviewing** has been ^und to be useful. Qualitative analysis 
of resul'ting evaluative data can be accqmpl Ishcd through **content analysis'* 
(Merton, 1968). "Discriminant analysis" (Tatsuoka, 1.970) has been found 
particularly useful in quantitative analysis of resulting evaluative data. 

• 12 
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Conclusion , 

At the beginning of this p^per,- U was noted that adult education was 
enjoying the limelight. It was suggested that sooner or later, to remain 

In the limelight, adult educators Will need to establish tj\e worth of 

. ' ' ' 1 ' ' ' " ^ 

their progrdms" through valid evaluation, particularly if those programs 

are publicly financed and broad-aimed./ The need to develop andjmplement 



/more adequate ways, to evaluate broad-aimed -adult education programs was 
/ describe^, along with suggestions for dolo.g so' through the use of **mlxed 
scanning.** ^ 

Developing '^and implementing innovative adult education programs can 
be challenging and rewarding. Perhaps mdre challenging Is developing and 
implementing appropriate ways of evaluatinlg those programs so that we pan 
learn from our experiences and demon s t ra t ej the^ wo ft h o f our programs to a 



public that seems currently to be favorably Incl ined^ £o*^SdCil t educatrtonv- 
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